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Abstract. We demonstrate that atmospheric neutrino data accumulated with the AMANDA 
and the partially deployed IceCube experiments constrain the allowed parameter space for 
VO a hypothesized fourth sterile neutrino beyond the reach of a combined analysis of all other 

experiments. We also illustrate the sensitivity of the completed IceCube detector, that is 
now taking data, to the parameter space of 3+1 model. 
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1 Introduction 

Data from atmospheric, solar, accelerator and reactor neutrino oscillation experiments have 
firmly established the pattern of oscillation of the three flavors of neutrinos [1]. The origin of 
oscillation is the non-vanishing masses and non-trivial mixing of the neutrino states. The 3v 
scheme of the neutrino sector of Standard Model has established the existence of two mass- 
squared differences Am s 2 ol = Am^ ~ 8 x 10 _5 eV 2 and Am 2 tm = lAm^j ~ |Am 2 2 | ~ 2 x 
10 -3 eV 2 related, respectively, to solar and atmospheric neutrino flavor oscillations (Am| = 
mf — rrij). The three mixing angles in the 3v scheme have also been measured: from solar [3] 
and KamLand [2] data sin 2 9\2 — 0.3; from atmospheric [4] and MINOS [5] data sin 2 #23 — 0.5. 
Recently the Double-CHOOZ [6], RENO [7] and Daya-Bay [8] experiments established a non- 
zero value for the last mixing angle sin 2 #13 ~ 0.1. 

Over the years a variety of experimental results have challenged the 3v framework, al- 
though none with completely convincing statistics. The simplest extension of the 3v frame- 
work to accommodate these anomalies is the so-called 3 + 1 scheme which assumes the 
presence of a fourth, mostly sterile, neutrino state with mass 777.4. The first challenge came 
from the LSND experiment in a search for the appearance D e in a beam over a base-line 
L ~ 30 m and neutrino energy 20 MeV < E u < 200 MeV [9]. The possible observation 
of i>a —> v e by LSND can be accommodated by intorducing a new mass-squared difference 
Am 2 -, — 0.2 — 10 eV 2 . The MiniBooNE experiment with base-line L ~ 540 m and neutrino 
energy 200 MeV < E u < 1.25 GeV and, therefore, the same L/E u as LSND, found sup- 
porting evidence at the 3.8<r level for the LSND result in both — > v e and — > v e search 
channels [10]. 

Recently the re-evaluation of the v e flux from reactors resulted in a 3.5% increase in the 
expected flux [11, 12]. With this new reactor v e flux, the combined results of all short base- 
line reactor neutrino experiments show a 3a deficit in the observed versus predicted number 
of events, the so-called "reactor anomaly" [13]. It can also be interpreted as evidence for a 
new mass eigenstate with Am^ ~ leV 2 . The observed 2.7a deficit in the expected number 
of events in the calibration of the solar neutrino experiments GALLEX and SAGE, the so- 
called "gallium anomaly" [14], also hints on the presence of a new mass-squared difference 
of the same order as reactor anomaly. The combined reactor and gallium anomalies exclude 
the no-oscillation hypothesis at 3.6a confidence level [13]. 

A hint on the presence of extra relativistic degrees of freedom also emerged from cosmol- 
ogy; the so-called "dark radiation" [15-17] (also see [18] and references therein). Although 
the significance of these hints depends on the data sample used in the analysis and on the 
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assumptions regarding the cosmological models, various analyses favor the presence of light 
sterile neutrinos at ~ 2a level with mass-squared difference Am^ ~ 0.1 eV 2 . 

While not compelling at the moment, it is imperative to test the evidence and one 
opportunity is provided by investigating the effects of light sterile neutrinos on the atmo- 
spheric neutrino flux. The partial propagation of atmospheric neutrinos through the Earth 
in a sterile state significantly modifies by the matter effects. With a mass-squared difference 
Ara^j ~ 1 eV 2 matter effect induces a MSW active-sterile resonance flavor conversion of the 
I'll iyu for < ) at the energies ~ TeV [19, 20]. The resonance leads to a distortion 

of the observed zenith angle distribution of atmospheric neutrino events. The construction 
of the km 3 -scale neutrino detector IceCube at the South Pole, sensitive to neutrinos in this 
energy range, has opened a window to search for the telltale distortions in the spectrum of 
atmospheric neutrinos. Specifically, atmospheric neutrino data taken with IceCube-40 [21] 
(i.e. the half-completed IceCube-86 detector), have raised the possibility of obtaining im- 
proved fits to the zenith angle distribution within the 3+1 scheme, or, alternatively, using 
the measurement to constrain its parameter space [22-25]. 

In this paper we revisit the information that IceCube can provide on the possible exis- 
tence of sterile neutrinos. Key is that our analysis combines the atmospheric data obtained 
by AMANDA-II over 6 years [26] with those of IceCube-40 [21]. Analyses like this are at best 
illustrative because of the role of systematic errors which have to be evaluated by the exper- 
iments. Combining data from two different experiments may partially remedy the problem. 
Our main conclusion is that the data do not support the existence of a fourth neutrino in 
the eV and sub-eV mass range and, on the contrary, yield limits on its parameter space that 
are stronger than those obtained from a combined analysis of all the other data [27]. We 
also present an estimate of the reach of the completed IceCube detector now taking data. 
Although the final sensitivities require an analysis that can only be performed by the individ- 
ual experiments, we are confident that the relative sensitivities, discussed here, are reliably 
estimated. 

After introducing the oscillation pattern in the presence of a fourth neutrino in section 2, 
we confront it with the atmospheric neutrino data of AMANDA and IceCube-40 in section 3. 
Our conclusion will be given at section 4. 

2 Oscillation probability in 3+1 scheme 

The 3+1 scheme for neutrino masses consist of three mostly active neutrinos with masses 
(mi, m,2, TO3) that accommodate the observation of solar and atmospheric oscillations, and a 
mostly sterile state with mass m.4 separated from active states by Am^ ~ 1 eV 2 3> Am^ 31 . 
In this scheme, due to the large Am^ and small active-sterile mixing, the effect of the sterile 
neutrino on the solar neutrino oscillation and conventional atmospheric neutrino oscillation 
(E u ~ GeV) is negligible. However, the new large mass-squared difference induces an active- 
sterile oscillation at short base-lines ~ 10 m for neutrinos with energy E v ~ 100 MeV, which 
is invoked to interpret the LSND, MiniBooNE, reactor and gallium anomalies [27-35]. 

The addition of one neutrino state leads to a generalization of the PMNS matrix to a 
4x4 unitary matrix and introduces three new mass-squared differences Am 2 ^ (i = 1,2,3). 
However, since Am^ 3> Am 2 131 , the 3+1 model effectively introduces four new parame- 
ters to the oscillation phenomenology: one mass-squared difference Am^ and three angles 
(#14; #24; #34) describing the active-sterile mixing. Assuming that all the CP-violating phases 
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vanish, the 4x4 unitary mixing matrix U4 can be parametrized in the following way [36]: 



U 4 = R- (6 | 34)R- (^24)R- (^i4)R- (^23)R (^i3)R (^12) , (2.1) 

where R 1J (#jj) (i, j = 1, . . . , 4 and i < j) is the 4x4 rotation matrix in the ij'-plane with the 
angle with elements 



r ij (%: 



= (SikSii+SjkSji)cij+(dikSji-SuSjk)sij+[(l - 5ik){l - Sji) + (1 - 5u)(l - 5jk)} Ski , 

M 

(2.2) 

where = cos 9{j and s^- = sin Oij . 

The atmospheric neutrinos propagate through the Earth before detection at up-going 
zenith angles at the South Pole. The evolution of the neutrino flavors inside the Earth is 
described by the following equation (a, /3 = e, (J,, r, s) 



ar 



-^U 4 M 2 U4 f + V(r) 



1 

Vf3 ~ -Taa^a , (2.3) 



J a/3 

where M 2 is the mass-squared differences matrix given by 

M 2 = diag (0, Amjj, Am^, Am^) . (2.4) 

The diagonal matrix V(r) is the matter potential as a function of distance r given by 

V(r) = V2G F diag(A^ e (r),0,0,iV n (r)/2) . (2.5) 

Here N e (r) and N n (r) are the electron and neutron number density of the Earth which we 
have fixed using the PREM model [37]. The last term in Eq. (2.3) takes into account the 
absorption of neutrinos inside the Earth; and the absorption matrix T is given by 

r = dia g (r e ,r At ,r T ,o) , (2.6) 

where r a = a UaP N p (r) + Cv a nN n (r). Np(r) and N n (r) are the proton and neutron number 
densities of the Earth and (T Vafl ^ is the total (charged current + neutral current) interaction 
cross section of v a with neutrons (protons). The neutrality of Earth implies that N e {r) = 
N p (r) and, to a good approximation N n (r) = N p (r): departures are less than 3%. Also, 
since the absorption of neutrinos inside the Earth is important for E v > 10 TeV, the charged 
lepton mass effect in the charged current cross section is negligible and we can assume that 
r e = = r T = r. The same equations as Eq. (2.3) describe the propagation of anti- 
neutrinos after replacing the cross section for neutrinos with those for anti-neutrinos and 
changing the sign of V(r). 

The flux of atmospheric neutrinos with energies E v > 100 GeV is mainly composed of 
f M and Vp,. The v e and v e flux is more than one order of magnitude smaller. The v t {Pt) flux 
is negligible up to energies of E u ~ 100 TeV where the charm contribution may surpass the 
conventional atmospheric flux. The onset of this flux has not been observed. IceCube detects 
the Cherenkov radiation from muons produced in charged current interaction of and 
with the nuclei inside or in the vicinity of detector. Thus, in our analysis we consider the 
survival probability of muon (anti-)neutrinos. The baseline of up-going atmospheric neutrinos 
detected by IceCube detector varies from zero for horizontal neutrinos to the diameter of the 
Earth 2i? e ~ 1.3 x 10 4 km for vertical up- going events. For neutrinos with the energy 
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E v > 100 GeV, the flavor oscillations resulting from the Am^ and Am^ mass splittings are 
negligible and the survival probability of muon (anti-)neutrinos in the 3+1 scheme in vacuum 
is given by 



where sin 2 29^ = 4|[/ /i 4| 2 (l — |[/^4| 2 ). With the parametrization chosen in Eq. (2.1) we 
have = ci4S24- The short base-line appearance experiments MiniBooNE {y^ —> v e and 
&n &e) an d LSND (pp — > v e ) are sensitive to sin 2 29 efM = 4\U e 4\ 2 \U fl 4\ 2 , while possible disap- 
pearance of the reactor v e flux constrains the mixing parameter sin 2 29 ee = 4|[/ e 4| 2 (l— |C/ e 4| 2 ). 
However, the three parameters (9 ee ,9 eiJi ,9 fll j i ) are not independent and the limits on two of 
them can be translated to the other one; see Refs. [33-35] for the global analysis and the 
interdependency of the mixing parameters. 

Although the vacuum survival probability for muon (anti-)neutrinos only depends on 
the 0i4 and #24, because of the critical role of matter effects through MSW resonant flavor 
conversion in our analysis, the survival probability also depends on the #34. This can be seen 
by writing the evolution equation Eq. (2.3) in the so-called propagation basis where the total 
Hamiltonian is diagonal. To illustrate the point, consider the evolution equation written in 
the basis (u e , v' v' T , v' s ) T = R 34 R 24 (z^ e , u^, u T , v s ) T . It is easy to check that the presence of 
the matter potential V and its nonzero element N n , through the term R 34 R 24 V R 24t R 34t , 
results in the dependence of the survival probability on #34 (for the details see Ref. [23]). Also, 
taking into account the current upper limit on the sin 2 29 ee = sin 2 29±4 from the reactor 
disappearance experiments, the dependence of the survival probability on 614 is very weak. 
We therefore make the approximation that 614 = 0, which implies 9^ = #24- 

Fig. 1 shows the oscillogram of the survival probability for different values sin 2 29^ 
and for fixed value Am^ = 1 eV 2 and #34 = 0, taking into account the absorption inside 
the Earth. In each oscillogram the axes are neutrino energy and the cosine of the zenith 
angle at IceCube. With increasing values of sin 2 29^ the presence of a dip in the survival 
probability develops for neutrino energies ~ TeV. The decrease in the survival probability 
at higher energies and vertical directions {cos9 z ~ —1) comes from the attenuation of the 
neutrino flux by the Earth (the last term in Eq. (2.3)). Also, for near vertical directions 
(cos #2 ~ [—1, —0.8]) the parametric resonance resulting from the propagation of neutrinos 
through the alternating mantle-core-mantle densities inside the Earth, will play a role [38]. 
To compare the survival probabilities for and iv, in Fig. 2 we show the oscillograms 
of Piy^ — >• fn) for the same mixing parameters as Fig. 1. For the v„ survival probability 
there is no MSW effect and therefore no dip. The absence of the MSW resonance for z/„ is a 
consequence of the normal hierarchy that we assumed between the active and sterile neutrino 
masses (Am|j > 0). For the inverted hierarchy (Am|j < 0) the resonance flavor conversion 
occurs for the channel and not for v^. However, from cosmological considerations, the 
Am^j < is strongly disfavored, because, in this case, the three active neutrino masses will 
all be at the ~ eV scale. In our analysis we consider the normal case (Awiy > 0) only. 

3 Parameter constraints from IceCube-40 and AMANDA-II data 

The number of events expected in IceCube can be calculated by the convolution of the 
neutrino flux with the effective area A e $ of the detector: 




(2.7) 



N = T(2vr) / A u cS (E u ,cos9 z )<f> u {E u ,cos9 z )dE u dcos9 z + [y -> v) 



(3.1) 
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Figure 1: The oscillogram for the survival probability P{p„ — > u^) for Am^ = leV 2 , 
sin 2 #34 = and the different values of sin 2 29^^ indicated in each sub-caption. The survival 
probability includes the attenuation of the neutrino flux inside the Earth from the charged 
and neutral current interaction with nuclei. 



where T is the livetime of data-taking and the 2tt factor comes from the integration over 
the azimuthal angle. The ^ U ^(E y , cos 2) is the flux of atmospheric (anti-)neutrinos as a 
function of neutrino energy and zenith angle in units GeV -1 m~ 2 s _1 sr _1 . The conventional 
atmospheric neutrino flux comes from the decay of pions and kaons produced in the interac- 
tions of cosmic rays with the Earth's atmosphere. From a few GeV to ~ 100 TeV this flux 
dominates and we will parametrize it following its calculation by [39]. For energies > 100 TeV 
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Figure 2: The same as Fig 1 but for P(vn —> Vp)- 



the contribution from the decay of charmed mesons and baryons may play a role, the so-called 
prompt flux, and we use the calculation of [40] to represent its possible contribution in this 
analysis. 

We analyze the data of the IceCube-40 and AMANDA-II experiments. The IceCube-40 
experiment measured the atmospheric neutrino flux in the energy range 100 GeV— 400 TeV 
with a livetime of T = 359 days; we use the effective area (separately for neutrinos and anti- 
neutrinos) in ten bins of cos #2 (with bin width 0.1) and 12 bins of E v (with bin width 0.3 
in log 10 (-Ev/GeV)) [21]. The AMANDA-II experiment measured the atmospheric neutrino 
flux in the energy range 10 5 — 10 6 GeV for 1387 days of data-taking [41, 42]. We used 
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the effective area (separateiy for neutrinos and anti-neutrinos) in ten bins of cos 6 Z (with bin 
width 0.1) and 25 bins of E u (with bin width 0.18 in log 10 (^/GeV)) [43]. The complete set 
of effective areas we used in our analysis are shown in Figs. 4 and 5 at the end of paper. 

In the calculation of the number of events using Eq. (3.1), the effective area is convo- 
luted with the atmospheric flux of neutrinos at the surface of Earth. This means that the 
attenuation of the neutrino flux inside the Earth is encoded in the effective area. But, in the 
presence of sterile neutrinos at 3+1 model, the attenuation of the neutrino flux is different 
from the standard 3v framework due to the singlet nature of sterile neutrinos. In the 3v 
framework and for energies E v > 100 GeV the Y term in Eq. (2.3) is flavor blind (diagonal) 
which enables us to factor out the attenuation term from the evolution equation. In this case 
the attenuation can be incorporated into the probability by a multiplicative exponential fac- 
tor exp[rX(cos 6 Z )], where X(cos6 z ) is the slant depth as a function of zenith angle. In order 
to take into account the difference in the absorption of neutrinos inside the Earth between 
3v and 3+1 models, we divide the effective area in each bin by the averaged attenuation 
exponential factor for 3u model in that bin. Then, in the calculation of number of events in 
Eq. (2.3), we insert the probability of oscillation including the absorption for the 3+1 model. 

To illustrate the reach of the IceCube-40 and AMANDA data to constrain the 3+1 
model, we define the following x 2 function 

X Am 41 , 6*34,6*245 a = > 5 + 2 ' ( 3 - 2 

where jV^ ata is the observed number of events in i th bin of cos^ and Nf +1 is the expected 
number of events in the i th bin of cos 9 Z assuming the 3+1 model with the mixing parameters 
(Ara^, 634, #24). The cr, = yj jydata j n ^ e denominator represents the statistical error in the 
observed events. The factor a is the normalization factor that represents the uncertainty in 
the normalization of the atmospheric neutrino flux with a a = 0.3 . 

Fig. 3 shows the 3a confidence level allowed region in the (Am^, sin 2 29^) plane from 
the combined analysis IceCube-40 and AMANDA-II data (red dashed curve). The best-fit 
to the data corresponds to the 3v framework. Introducing a sterile neutrino deteriorates the 
fit (increases the x 2 value) . The black solid vertical line shows the upper limit on the mixing 
angle (independent from the mass-splitting value) which comes from the non-observation 
of the oscillation pattern in the low energy down-going atmospheric neutrinos which leads 
to requirement of large mixing elements corresponding to light mass states (i. e. large 
5^i=i l^/iil 2 ) [44]- The green solid curve shows the upper limit from the disappearance data 
at 3o" confidence level (including reactor anomaly, MINOS and CDHSW) [35]. In the range 
Am 41 ~ 0.1 — 1 eV 2 relevant for the interpretation of the recent experimental anomalies, 
the combined IceCube and AMANDA data strongly constrain the parameter space. The 
blue dotted curve shows the prospect for the complete IceCube detector to constrain the 
3+1 model after three years of data-taking. However, it should be mentioned that for the 
sensitivity of IceCube-80, we have used a coarse binned non-optimized effective area [45, 46] 
and the sensitivity of the full IceCube80+DeepCore should be significantly better. The 
weaker constraint for larger values of Am^ results from the fact that by increasing the value 
of the mass-splitting the dip in the survival probability shifts to higher energies; where the 
flux of atmospheric neutrinos is reduced and the statistics worse. This will be remedied by 
future data from the completed detector. For the Am^ ~ 0.1 — 1 eV 2 the resonance occurs 
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Figure 3: The allowed region in the (Am^sm 2 20„„) parameter space from the combined 
analysis IceCube-40 and AMANDA-II data (red dashed curve) at 3a confidence level. The 
green solid curve shows the 3a upper limit from the disappearance data taken from Ref. [35] . 
The blue dotted curve shows an estimate of the 3a sensitivity of IceCube-80 after three years 
of data-taking. The black vertical line shows the upper limit from low energy atmospheric 
neutrino oscillation data [44]. 

at 0.3 — 3 TeV, where the flux of atmospheric neutrinos is sizable and the detection efficiency 
is optimized. 

4 Summary and Conclusion 

The presence of a fourth sterile neutrino state leads to a distortion of the zenith angle 
distribution of high energy atmospheric neutrinos through the MSW active-sterile resonance 
inside the Earth. The energy range of the resonance is ideally covered by the South Pole 
detectors, IceCube and AMANDA. The presence of this resonance is not supported by the 
present evidence. We have shown that the 3+1 model is severely constrained. The limit 
obtained in our analysis is stronger than the combined limit of all other experiments. Also, 
we have shown that the completed IceCube detector can test the 3+1 model for mixing angle 
as small as sin 2 29^ ~ 10~ 2 . The complete detector has already collected data for more 
than one year and IceCube will therefore further constrain the presence of sterile neutrino, 
or possibly discover it. 
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IceCube— 40 effective area 



IceCube— 40 v t , effective area 




Figure 4: The A^ s and for IceCube-40 experiment as a function of neutrino energy 
and zenith angle. The vertical dashed grid lines show the binning in energy. The figures 
correspond to: (a) neutrino effective area for —0.5 < cos# z < 0; (b) anti-neutrino effective 
area for —0.5 < cos6* 2 < 0; (c) neutrino effective area for —1 < cos6> 2 < —0.5; (d) anti- 
neutrino effective area for —1 < cos# 2 < —0.5 . 
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Figure 5: The same as Fig. 4 but for AMANDA-II experiment [43]. 
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